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ABSTRACT 



The Learning Equation (TLE) is a software product that 
provides mathematics programming to students. It was introduced to grade 9 
students in Alberta in 1997-1998, and in 1998-1999, its use was extended to 
grades 7, 8, and 10. TLE is designed to follow the Alberta mathematics 

curriculum closely. It can be implemented in several ways, and in some 
schools is the main delivery method, entirely replacing textbooks. In other 
schools, it is a supplement or a remedial resource. This study examined the 
effects of TLE on student achievement and attitudes and on teachers 1 
effective delivery of mathematics. Interviews were conducted with 1,186 
students, and data, including achievement findings, were available from 
several sources. Overall, by grade 9, students who used TLE slightly 
outperformed those who did not. TLE did not appear to make students feel more 
positive about mathematics, but in some cases, it reduced negative feelings. 
Teachers reported positive attitudes about TLE, but wished for closer 
communication with the TLE development community. In general, study findings 
reflect positively on TLE and indicate that it is a viable alternative to 
teaching mathematics using a textbook. Three appendixes describe study 
methodology and present pretest and posttest student responses. (Contains 17 
tables and 23 graphs.) (SLD) 
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Executive Summary 
Introduction 

The Learning Equation (TLE) is a software product that provides mathematics programming to students. It 
was introduced at the Grade 9 level in Alberta in 1997/98. In 1998/99 its use was extended to Grades 7, 8 
and 10. TLE was developed in cooperation with Nelson Canada; Alberta Learning provided both financial 
support and developmental /curricular assistance. TLE closely follows the Alberta mathematics curriculum. 

TLE can be implemented in a variety of ways. In some schools, for example, TLE is used as the mam 
delivery method, entirely replacing textbooks. In others it is used as a supplementary (i.e. used for selected 
units) or remedial resource. Though TLE was not mtended for use as a stand-alone program for virtual 
education, it appears that it is bemg used that way m some schools. 

In 2000, Alberta Learning initiated a study to examine the impact of TLE on students and teachers. This 
study examined the following three goals of TLE: 

• Increase student achievement m mathematics 

• Positively influence students’ attitudes towards mathematics 

• Assist teachers with the effective delivery of mathematics. 

Each of the three goals was examined separately through analysis of student achievement and attitudes and 
TLE teacher’s experiences. Both quantitative and qualitative research methodologies were used to explore 
the impact of TLE on student achievement and its effect on students’ perceptions of mathematics and 
teachers’ satisfaction with the effectiveness of TLE in assisting with mathematics instruction. 

Relationship to Alberta Learning’s Business Plan 

This project relates to Alberta Learning’s business plan in two key ways: 

Goal 1: High Quality Learning Opportunities 

• Outcome: Responsiveness and Flexibility: The learning system is flexible and provides a variety of 
programs and modes of delivery 

Goal 5: Highly Responsive and Responsible Ministry 

• Outcome: The Ministry demonstrates leadership and continuous improvement in administrative and 
business processes and practices 
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Summary of Findings 

Student Achievement 

The 1996/97 Grade 6 and 1999/00 Grade 9 Mathematics Provincial Achievement Tests were used to 
examine the achievement of students who used TLE in Grade 9 and those who did not. Overall, students 
who used TLE in Grade 9 slighdy outperformed their non-TLE counterparts. 

The following table presents the 1996/97 Grade 6 and 1999/00 Grade 9 Mathematics Achievement Test 
results by TLE usage. In interpreting these results, it should be noted that students are divided into TLE and 
non-TLE groups based on their mode of instruction in Grade 9 — neither group used TLE in Grade 6. At the 
Grade 6 level the two groups were very similar. Approximately equal proportions of the TLE (90.9%) and 
non-TLE groups (91.2%) achieved the acceptable standard on the Grade 6 Mathematics Achievement Test. 
Twenty- six percent of each group achieved the standard of excellence. The mean scores 1 of each group were 
also remarkably similar — 35.2 for the TLE group and 35.1 for the non-TLE group. The difference between 
the groups 5 Grade 6 mean scores was not statistically significant. 

By Grade 9, the achievement of the TLE and non-TLE groups is different. The TLE group has a 
significantly higher mean score and a significantly greater proportion of students achieving the standard of 
excellence. The two groups are not significandy different at the acceptable standard. Overall, the 1.2 raw 
score point difference in the average scores is statistically significant and indicates that TLE students 
outperformed their non-TLE counterparts. 



Mathematics Achievement Test Results 




Grade 6: 1996/1997 


Grade 9: 1999/2000 


TLE Group 


Non-TLE 

Group 


Province 


TLE Group 


Non-TLE 

Group 


Province 


Number 


1 658 


21 874 


38 764** 


1 658 


21 874 


37 419** 


Mean Score 


35.2 


35.1 


34.1 


31.4* 


30.2* 


30.0 


Standard 

Deviation 


8.5 


8.4 


9.0 


10.5 


10.1 


10.3 


% Achieving 
Acceptable 


90.9 


91.2 


87.7 


76.7 


75.0 


73.6 


% Achieving 
Excellence 


26.2 


26.2 


24.4 


20.3* 


15.7* 


15.7 


^Difference between TLE and non-TLE groups is statistically significant, p<.01 

**The TLE and non-TLE groups are not equal to the total number of students in the province because some 
students were omitted from the analysis. Please see Appendix 1 for and explanation of the omissions. 



1 The maximum possible score for these 2 mathematics Achievement Tests was 50. 
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The major component of the student achievement analysis involved using linear regression to predict 
students’ 1999/00 Grade 9 Mathematics Provincial Achievement Test score based on their 1996/97 Grade 6 
Mathematics score 2 . This technique is useful because it takes into account the prior achievement level of 
students when analyzing' current results. Grade 6 mathematics scores of non-TLE students were used to 
predict what a student would be expected to achieve in Grade 9. This, in effect, meant that non-TLE 
students were used as a control group. The results of this analysis showed that the TLE students sigmficandy 
outperformed expectations. As the following table shows, TLE students scored 1.04 raw score points higher 
than predicted 3 . 



Differences in Achievement Test Raw score points 
Using/Not Using TLE 




Number 


Predicted Grade 9 
Mean 


Actual Grade 9 
Mean 


Difference 
(Actual - Predicted) 


Using TLE 


1 658 


30.32 


31.36 


1.04* 


Not using TLE 


21 874 


30.24 


30.24 


N/A 


*p<0.01 



Student Survey 

Junior high students using TLE for the first time, and a control group who did not use TLE, were surveyed 
about their attitudes towards mathematics in September 2000 and then again in May 2001. Survey results 
were analyzed according to three concepts — confidence in mathematics ability, attitudes towards mathematics 
and motivation. 

While results are somewhat mixed, the experimental group appears to have obtained some “buffering” effects 
from TLE. While TLE does not appear to have caused students to feel more positive towards mathematics, 
it seems that it did, in some cases, help to prevent them from feeling more negative. While motivation 
decreased across all groups, confidence in mathematics ability and attitudes towards mathematics were 
maintained in the TLE group. Only amongst Grade 7s using TLE were decreases seen across all scales. 



Confidence, Attitude and Motivation Changes in TLE and Non-TLE Students 




TLE 


Non-TLE 




Confidence 


Attitudes 


Motivation 


Confidence 


Attitudes 


Motivation 


Overall 


No change 


No change 


Decrease 


Decrease 


Decrease 


Decrease 




Grade 7 


Decrease 


Decrease 


Decrease 


No change 


Decrease 


Decrease 


Grade8 


No change 


No change 


Decrease 


Decrease 


Decrease 


Decrease 


Grade 9 


No change 


No change 


Decrease 


No change 


No change 


Decrease 




Females 


No change 


No change 


Decrease 


No change 


Decrease 


Decrease 


Males 


No change 


No change 


Decrease 


Decrease 


Decrease 


Decrease 



In addition to comparing results between TLE and non-TLE students, the survey also .obtained feedback 
about the program from TLE students. This analysis showed that students who used TLE more frequendy 
(more than 3 times/week) were more likely to have positive attitudes towards mathematics, computers and 
TLE, itself than did students who used TLE less frequendy (less than 3 times/week). 



2 See the “Student Achievement Analysis” section of Appendix 1 for a detailed description of the methodology. 

3 The maximum possible score on this mathematics Achievement Tests was 50. 
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Teacher Focus Groups 

Overall, teachers were very positive about their TLE experiences and thought their students benefited from 
this new form of instructional delivery. Many perceived positive attitudinal changes in their students. In 
addition, teachers thought they benefited from the opportunity to employ creative and innovative teaching 
strategies and their students also benefited from this. Focus group participants thought TLE allowed 
students to work at their own pace and provided teachers with an opportunity for more one-on-one and small 
group instruction. The vast majority of focus group participants agree TLE works best for mid-level 
students. There was some concern that it was not ideally suited to other groups such as below average and 
modified program students. 

While teachers were generally happy about what is occurring in their classroom, many thought additional 
supports are needed. Almost uniformly, teachers thought both Alberta Learning and Nelson could facilitate 
more effective communication within the TLE community. In part, this should include the key component 
of receiving in-service prior to implementing TLE; a factor many participants thought was necessary for 
successful TLE implementation. Teachers expressed the desire to maintain a close, continuous relationship 
with the software developer. 

Teachers were also careful to point out the key role that teacher motivation plays in TLE implementation. In 
terms of a teacher making a decision to start with TLE, the focus group participants thought using TLE “is 
not for everyone.” In a few cases, teachers were directed to use TLE by the school administration. Not 
surprisingly, this seemed to increase the chances of having a negative TLE experience. 

Another important aspect of TLE implementation is having adequate technology. Schools without adequate 
technical support were finding TLE implementation very difficult. For those who had adequate support, 
however, TLE offered an opportunity to bring a new and exciting form of learning to the classroom. 

Observations 

Some key observations that should be considered when implementing TLE were noted during the course of 
the study: 

• Administrative support at both the school and school authority levels is crucial for successful TLE 
implementation. This is especially important in terms of ensuring sufficient technical resources are 
available. 

• TLE may be particularly useful to non-specialist math teachers because it can complement their 
instruction with another perspective. 

• The minimum technical standards specified for TLE may not be acceptable to many students - 42% of 
grade sevens, 37% of grade eights and 42% of grade nines thought that their computers were too slow. 

In addition, consideration should also be given to the reading ability of students. The amount of reading 
required by TLE may be difficult for some students and suggests that TLE could be improved by additional 
audio support (in order to decrease the visibility of students who need to use audio support, some teachers 
suggested that all students be allowed to bring CDs into class). 
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Recommendations 

For schools/ school authorities: 

1. Teacher support is obtained prior to TLE implementation. Efforts to implement TLE without full 
teacher support were generally not positive. 

2. Teachers are provided with the opportunity to attend in-service prior to TLE implementation. Many 
teachers found this to be an extremely useful part of TLE preparation. 

3. Schools should have sufficient technical infrastructure prior to TLE implementation. This allows teachers 
and students to focus on mathematics without the distractions and delays caused by crashing computers, 
etc. 

For Alberta Learning: 

4. Alberta Learning support the provision of information about TLE resources to teachers in order to 
facilitate better communication with the TLE community. 

5. Alberta Learning continue to support the development of TLE. 

6. This study be made available to school and school authority staff to inform programming decisions. 

Future Research 

The current study provides a broad overview of the relationship between TLE and student achievement and 
attitudes. It cannot, however, definitively isolate the unique effect of TLE on these factors. While this study 
showed an overall increase in achievement by TLE students, additional information on student characteristics 
would be helpful in interpreting these achievement gains — do all, or only some types, of students benefit 
from using TLE? Future research may want to focus on a more detailed investigation of TLE use in the 
classroom. Investigation of successful models of TLE implementation could be particularly useful in 
developing “best practices” guidelines. 

Conclusion 

In general, the overall results of this study reflect positively on TLE. The results of this study suggest that 
while TLE is not the answer to all mathematics issues, it should be considered a viable alternative to teaching 
mathematics using a textbook. For one, TLE offers an innovative way for students to learn and teachers to 
teach mathematics. In addition, TLE students in this study felt less negative about Mathematics than non- 
TLE students. Lastly, while this study cannot answer all of the questions surrounding the relationship 
between TLE and student achievement, it does suggest that, at a general level TLE may positively impact 
achievement for some students. 
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Student Achievement Analysis 

Introduction 

A key goal of this study was to examine the role of TLE in student achievement. In order to determine the 
impact that TLE has had on student achievement, linear regression, was used to predict students’ 1999/00 
Grade 9 Mathematics Provincial Achievement Test (PAT) scores based on their 1996/97 Grade 6 
Mathematics PAT scores. This technique examines students’ current achievement net of their prior 
achievement. Only students with both 1996/97 Grade 6 and 1999/00 Grade 9 Mathematics PAT scores 
were included in the analysis. 

Students were assigned to TLE and non-TLE groups based on information provided by Nelson Canada and 
obtained through telephone interviews with school staff. Those schools where students may have been using 
TLE were classed as “indeterminate” and largely removed from the analysis. The 1996/97 Grade 6 
(independent variable) and 1999/00 Grade 9 (dependent variable) Mathematics PAT scores of students who 
were not using TLE were used to generate a regression equation. This equation was then used to generate 
estimated Grade 9 scores for each TLE student using their actual 1996/97 Grade 6 Mathematics PAT results. 
In total, 1 658 students were included in the TLE group and 21 874 students were included in the non-TLE 
group. 

In interpreting the achievement results it is important not to confuse correlation and causation. While the 
results indicate that TLE students scored higher than expected, this does not automatically lead to the 
conclusion that the achievement gain was due to TLE. 

Grade 6 and 9 Mathematics Achievement Test Results 

Prior to running the predictive regression, descriptive statistics on the 1996/97 Grade 6 and 1999/00 Grade 9 
Mathematics PATs for the TLE and non-TLE groups were examined. The following table presents the 
Grade 6 and 9 Mathematics PAT results by TLE usage. For both groups, a lower proportion of students 
attain the acceptable standard and the standard of excellence on the grade 9 test than on the grade 6 test. At 
the Grade six level the two groups were very similar. Approximately equal proportions of the TLE (90.9%) 
and non-TLE groups (91.2%) achieved the acceptable standard on the Grade 6 Mathematics PAT. 26.2% of 
each gtoup achieved the standard of excellence. The mean scores 4 of each group were also remarkably similar 
- 35.2 for the TLE group and 35.1 for the non-TLE group. The difference between the groups’ grade 6 
mean scores was not statistically significant. 

By grade 9, the achievement of the TLE and non-TLE groups is different. The TLE students are 
outperforming their non-TLE counterparts at the standard of excellence by 4.6%. In addition, at the Grade 9 
level, the difference between the mean raw score points of the two groups (1.2) is statistically significant 5 . 



4 The maximum possible score for these 2 mathematics Achievement Tests is 50. 

5 Based on an independent samples t-test. 
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Mathematics Achievement Test Results 




Grade 6: 1996/1997 


Grade 9: 1999/2000 


TLE Group 


Non-TLE 

Group 


Province 


TLE Group 


Non-TLE 

Group 


Province 


Number 


1 658 


21 874 


38 764 


1 658 


21 874 


37 419 


Mean Score 


35.2 


35.1 


34.1 


31.4* 


30.2* 


30.0 


Standard Deviation 


8.5 


8.4 


9.0 


10.5 


10.1 


10.3 


% Achieving Acceptable 


90.9 


91.2 


87.7 


76.7 


75.0 


73.6 


% Achieving Excellence 


26.2 


26.2 


24.4 


20 . 3 * 


'15.7* 


15.7 


^Difference between TLE and non-TLE groups is statistically significant, p<0.0l 



Regression Analysis 

A predictive regression equation was generated based on the Grade 6 and 9 Mathematics PAT scores of 
students who did not use TLE. In general terms, the regression equation represents what the “average” 
student would be expected to accomplish. Each student’s grade 6 score is put into the equation to generate a 
predicted grade 9 score. A student’s predicted score is then compared to his/her actual score to determine if 
he/ she performed above, below or as predicted. 

The data collection and analysis procedures used to generate the regression equation were as follows: 

• The 1996/97 Grade 6 Mathematics PAT scores of students not using TLE were used to generate a 
regression equation to predict scores for the 1999/00 Grade 9 Mathematics PAT. This group served as a 
control group. 

• Using the generated regression equation, predicted scores for all students were calculated. The actual 
1999/00 Grade 9 Mathematics PAT scores of the TLE group were then compared to their predicted raw 
scores. (Note: Because students in the non-TLE group were used to generate the predicted scores, as a 
group their difference from the prediction is zero.) 

The non-TLE Grade 6 scores (the independent variable) explained approximately Vz (R 2 =0.512) of the 
variation in the Grade 9 scores (the dependent variable). Additional variables added to the regression 
equation — e.g. gender — did not substantially change the R 2 value, and, hence were omitted from the equation. 

The remainder of this section compares the actual results of the TLE group to their predicted results using a 
t-test to determine if there are significant differences. 

As the following table illustrates, students using TLE scored, on average, 1.04 raw score points higher than 
predicted. This translates to a gain of approximately 2%. 





Differences in Achievement Test Scores 
Using/ Not Using TLE 






Number 


Predicted Grade 9 
Mean 


Actual Grade 9 
Mean 


Difference 
(Actual - Predicted) 


Using TLE 


1 658 


30.32 


31.36 


1.04* 


Not using TLE 


21 874 


30.24 


30.24 


N/A 


*p<0.0l 
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Using TUE as a Primary Resource 

Teachers can use TLE in a variety of ways. To further explore the effect of TLE, teachers were asked to 
provide additional information on exacdy how they were using TLE in their classroom. One of the ways 
teachers can use TLE is as a “primary resource”, that is TLE (including the student refresher) is used instead 
of a textbook. Forty two per cent (N=697) of the sample used TLE as a primary resource. The actual Grade 
9 mean for this group of students was 1.20 raw score points above that which was predicted (a 2.4% gain). 





Table 3 

Using TLE as a Primary Resource 






Number 


Predicted Grade 9 
Mean 


Actual Grade 9 
Mean 


Difference Score 
(Actual — Predicted) 


Primary TLE use 


697 


30.04 


31.24 


1.20* 


*p<0.01 











Using TLE as a Supplementary Resource 

Fifty eight per cent (n=961) of our sample was using TLE as a “supplementary resource”: Students accessing 
TLE in this manner would typically be spending less time on TLE than those using it as a primary resource. 
Supplementary TLE use can be defined as follows: 

• Selected strands or lessons — TLE is used for all of a particular topic area (strand) or for all of the material 
in selected lessons within a strand. 

• Blended with textbook — some portion of the lesson is given using traditional delivery and some is 
delivered using TLE, often this group spends one or a few days a week in the lab. 

Those using TLE as a supplementary resource scored 0.93 raw score points above the prediction for their 
group. 





Using TLE as a Supplementary Resource 






Number 


Predicted Grade 9 

Mean 


Actual Grade 9 
Mean 


Difference Score 
(Actual - Predicted) 


Supplementary TLE use 


961 


30.52 


31.45 


0.93* 


*p<0.01 











Gender 

Analyzing the sample according to gender revealed positive TLE-related score differences for both males and 
females. Both male and female TLE users sigmficandy outperformed their non-TLE counterparts; in both 
cases the difference was approximately 1 raw score point. 



Gender Differences 
TLE /non-TLE 




Number 


Predicted Grade 
9 Mean 


Actual Grade 
9 Mean 


Difference Score 
(Actual — Predicted) 


TLE — non TLE 


Males using TLE 


855 


31.06 


31.98 


0.92 


1.19* 


Males not using TLE 


11 174 


30.88 


30.62 


-0.27 




Females using TLE 


803 


29.52 


30.70 


1.17 


0.90* 


Females not using TLE 


10 700 


29.57 


29.84 


0.27 


*p<0.01 
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